UDF to SQL translation through compositional lazy inductive synthesis

Many data processing systems allow SQL queries that call user-defined functions (UDFs) written in conventional programming languages. While such extensions provide convenience and flexibility to users, involving UDFs are not as efficient their pure counterparts invoke SQL’s highly-optimized built-in functions. Motivated by this problem, we propose a new technique for translating with expressions. Unlike prior work space, our method is based on syntactic rewrite rules can handle much more general class of UDFs. At high-level, counterexample-guided inductive synthesis (CEGIS) but employs novel compositional strategy decomposes the task into simpler sub-problems. However, because there no universal decomposition works all UDFs, lazy approach generates sequence decompositions correspond increasingly harder problems. Because most realistic UDF-to-SQL translation tasks amenable fine-grained strategy, scales significantly better than traditional CEGIS. We have implemented proposed tool called CLIS optimizing Spark programs containing Scala To evaluate CLIS, manually study 100 randomly selected find 63 them be expressed SQL. Our evaluation these shows automatically synthesize equivalent expressions 92% cases it solve 2.4× benchmarks compared baseline does use approach. also show yields an average speed-up 3.5× individual 1.3× 3.1× terms end-to-end application performance.